Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable kube-rbac-proxy from prometheus-exporter-operator controller-manager #26

Merged
merged 4 commits into from
Apr 19, 2021

Conversation

slopezz
Copy link
Member

@slopezz slopezz commented Apr 19, 2021

kube-rbac-proxy is used by default on operator-sdk to protect operator metrics path, in case you don't want anyone in the cluster but only k8s authenticated resources can access to them.

But there are 2 current issues with kube-rbac-proxy:

  1. Upgrade from kube-rbac-proxy:v0.5.0 to latest kube-rbac-proxy:v0.8.0 Ansible operator-sdk v1.5.0 with updated kube-rbac-proxy:v0.8.0 fails to run with permission denied operator-framework/operator-sdk#4684 causes error on OpenShift 4.6+:
Error: container create failed: time="2021-03-19T15:51:12Z" level=error msg="container_linux.go:366: starting container     process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  • The current solution is to use a different proxy image on OpenShift 4.6+ openshift4/ose-kube-rbac-proxy:v4.7.0 (which works OK), but this image is behind registry.redhat.io registry which requires authenticated if not using Openshift (it doesn't work directly on vanilla k8s), so you need to maintain 2 different bundles with different proxy images if you want to run the operator on both OpenShift or K8s, which makes maintenance more complex.
  1. In addition, OpenShift User Workload Monitoring (OpenShift official monitoring stack) does not support ServiceMonitor with bearerTokenFile field Prometheus ServiceMonitor failing to scrape operator metrics served though kube-proxy HTTPS 8443 port operator-framework/operator-sdk#4764 (comment) (which is needed to scrape metrics behind kube-rbac-proxy) , so it seems there is no way of having operator metrics with auth if using OCP UWM.

For that reason, taking into account that operator metrics are not that important to have them with forced auth, I have disabled kube-rbac-proxy container (making a few changes to make that work with a new patch, and leaving default proxy yamls there, in case they want to be enabled easily in the future), so anyone once inside the cluster could check operator metrics without any problem on both OCP and k8s (even if using OCP UWM).

@3scale-robot 3scale-robot added needs-kind Indicates a PR or issue lacks a `kind/foo` label and requires one. needs-priority Indicates a PR or issue lacks a `priority/foo` label and requires one. needs-size Indicates a PR or issue lacks a `size/foo` label and requires one. labels Apr 19, 2021
@slopezz
Copy link
Member Author

slopezz commented Apr 19, 2021

/kind feature
/priority important-soon
/label size/xs
/assign

@3scale-robot 3scale-robot added kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next sprint. size/XS Requires less than an hour to complete the PR or the issue. and removed needs-kind Indicates a PR or issue lacks a `kind/foo` label and requires one. needs-priority Indicates a PR or issue lacks a `priority/foo` label and requires one. needs-size Indicates a PR or issue lacks a `size/foo` label and requires one. labels Apr 19, 2021
@roivaz
Copy link
Member

roivaz commented Apr 19, 2021

/lgtm

@3scale-robot 3scale-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 19, 2021
@3scale-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 81832963d343714fa6597e71a85f79fc36b6afa0

@raelga
Copy link
Contributor

raelga commented Apr 19, 2021

/lgtm

@raelga
Copy link
Contributor

raelga commented Apr 19, 2021

/shrug

@slopezz
Copy link
Member Author

slopezz commented Apr 19, 2021

/approve

@3scale-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slopezz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@3scale-robot 3scale-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 19, 2021
@3scale-robot 3scale-robot merged commit 88759a2 into main Apr 19, 2021
@3scale-robot 3scale-robot deleted the disable-kube-proxy branch April 19, 2021 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/feature Categorizes issue or PR as related to a new feature. lgtm Indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next sprint. size/XS Requires less than an hour to complete the PR or the issue. ¯\_(ツ)_/¯
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants